DOCUMENT RESUME 



ED 409 181 



SE 060 342 



AUTHOR 

TITLE 

PUB DATE 
NOTE 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Clariana, Roy B. 

The Effects of an Integrated Learning System on Third 
Graders' Mathematics and Reading Achievement. 

94 
?P • 

Reports - Research (143) 

MF01/PC01 Plus Postage. 

Academic Achievement; Educational Strategies; *Grade 3; 
integrated Activities; Mathematics; *Mathematics 
Achievement; Primary Education; Reading; * Reading 
Achievement; Teaching Methods; Technology 



ABSTRACT 
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instruction. The California Test of Basic Skills Total Mathematics and Total 
Reading scores given at the end of the third grade served as the posttest . 

The data were analyzed by mixed multi-variate analysis of covariance. A 
significant main effect for ILS treatment, and a significant interaction 
between content and treatment were observed. Under ILS, larger gains occurred 
for mathematics than for reading. Possible reasons for this finding are 
presented and the implications of the findings are discussed along with 
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The Effects of an Integrated Learning System 
on Third Graders' Mathematics and Reading Achievement 



Roy B. Clariana 
Jostens Learning Corporation 

This study considers the effects of Jostens Learning Corporation's Integrated Learning System 
(ILS) on the mathematics and reading standardized test scores of elementary school children. It 
was anticipated that test scores would increase modestly with the introduction of the ILS, 
particularly in mathematics. The sample consisted of four consecutive intact classes (final 
sample n = 85) taught by the same third-grade teacher. The first and second classes received 
traditional classroom instruction (baseline, control group) while the third and fourth classes 
received traditional classroom instruction plus standard sequence ILS instruction (treatment 
group). The California Test of Basic Skills Total Mathematics and Total Reading scores given 
at the end of the third grade served as posttesL The data were analyzed by mixed multi-variate 
analysis of covariance (MANCOVA). A significant main effect for ILS treatment, and a 
significant interaction between content and treatment were observed. Under ILS, larger gains 
occurred for mathematics, es = 0.49 than for reading, es = 0.06 (unadjusted). Possible reasons 
for this finding are presented and the implications of the findings are discussed with suggestions 
for further research. 



Integrated learning systems (ILSs) are used by mil- 
lions of school children each year. A survey conducted by 
Market Data Retrieval (1991) indicated that ILSs are in- 
stalled in about 1 1 % of K- 12 school districts. Currently, ILS 
software purchases account for more than half of all educa- 
tional software dollars (Bailey, 1993; Trotter, 1990). 

Mary-Alice White (1993) describes several strengths 
of ILSs. She notes that ILSs provide mastery-learning based 
individualized interactive exposure to a systematic curricu- 
lum (usually mathematics, reading, and language arts) with 
comprehensive reports of student progress. Also, the "real" 
curriculum is available for critique, review, and revision. 

Problematically however, a useful research base re- 
lated to ILSs is practically nonexistent (Trotter, 1990). In a 
review of 30 studies of the effectiveness of ILSs, Becker 
(1992) noted that most were of an unsatisfactory quality 
(mostly methodological problems) and that no definitive 
conclusions could be obtained. Because of the extensive use 
of ILSs in American education despite the limited research 
base, additional ILS research and model building are criti- 
cally important. 

Is an ILS equally effective for both mathematics and 
reading? With an ILS, much thought has been applied to 
create a developmentally appropriate sequence of activities. 
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This may be especially important in a hierarchically ar- 
ranged content area like mathematics. Further, many mathe- 
matics concepts tend to be visual, and so may be easier to 
represent on a microcomputer (a visual medium) than con- 
cepts from reading comprehension or language arts. This 
does not mean that CBI is ineffectual in reading, only that 
CBI may be even more effectual with content areas like 
mathematics 

For example, one study utilized mathematics and lan- 
guage arts ILS lessons along with teacher-directed lessons in 
a six-weeks summer remedial program for eighth-grade 
students. Pretest to posttest effect sizes of 0.44 for language 
arts and 0.76 for mathematics were observed (Clariana & 
Schultz, 1993). Becker (1993) reported about equal gains 
forreading and mathematics though mathematics was slightly 
greater. Given that other variables are about equal (e.g., 
time-on-task, support, ...), it seems reasonable that ILS gains 
in mathematics may be greater than gains in reading. At any 
rate, further research into this possible CBI differential 
effect by content is warranted. 

Since the teacher is vitally important to student prog- 
ress and achievement via CBI (Clariana, 1990), it is impor- 
tant to control for teacher effects. Therefore, this study con- 
sidered the effects of an ILS on the standardized test scores 
of third-grade students taught by the same third-grade teacher. 
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This study addressed the following questions: (a) is there a 
difference for mathematics versus reading scores, and (b) 
what sorts of effect sizes are observed on this standardized 
test. 

Method 

Participants 

An initial sample of 115 third-grade students from a 
predominantly white, rural elementary school participated 
in the study. The initial sample consisted of four separate 
third grade classes taught by one teacher during a period of 
four consecutive years. Since district test policies provided 
for standardized testing at the end of both second and third- 
grades, these scores could be used to assess learning gains 
for the ILS implemented in the third grade. The final sample 
consisted of 85 of the original 115 students. Subjects were 
lost from the initial sample for various reasons such as 
incomplete test scores and lack of parental permission. 

Program 

Because of the powerful effects of an individual teacher 
on student achievement, the present study considered the 
effects of an ILS with one teacher. Two intact classes just 
prior to the ILS introduction were utilized as the baseline or 
control group, and two classes participating in the ILS 
served as the experimental group. Because of this lack of 
random assignment, pre treatment group equivalence was 
investigated by analysis of variance of standardized test 
scores (i.e., the California Test of Basic Skills) obtained 
during second grade. For reading, the control group (x = 
68.2) was not significantly different than the ILS treatment 
group {x = 64.8), F( 1,84) = 0.43. For mathematics, the 
control group (x= 68.3) again was not significantly different 
than the ILS treatment group (x = 64.4), F(l,84) = 0.57. 
Note that this amounts to a pre-treatment effect size differ- 
ence of es * 0.2 in favor of the control group. 

The third-grade teacher in this study had more than 15 
years of teaching experience in the district, most at the third- 
grade level. Also, this teacher used the same methods in the 
classroom with each class. During computer time, the 
teacher accompanied the students to the computer lab but 
was minimally engaged in instruction in the lab. 

The ILS instruction consisted of Jostens Learning 
Corporation's (JLC) reading and mathematics expansions 
lessons for third grade. The software used was well de- 
signed, addressing text factors suggested by previous re- 
search like high proportions of highlights, cues, and prompts; 
wider margins; double-spacing; and multiple levels of help 
screens (Allesi & Trollip, 1985; Bork, 1987; Grabinger, 
1983) as well as functional page formatting, item specific 
feedback, animation, extensive use of instructional graph- 



ics, and digitized audio (human-voice). Most importantly, 
the software automatically employed mastery-learning 
procedures with every activity, including additional reme- 
dial activities and sometimes direct teacher intervention 
(e.g., one-to-one tutoring) as necessary. 

The mathematics software consisted of 957 activities 
that provided approximately 27 1 hours of instruction from 
kindergarten through fourth grade levels. The activity 
developmental sequence corresponded to the major mathe- 
matics textbooks in terms of content and sequence. Activi- 
ties typically required 5 to 15 minutes to complete and were 
grouped together into lessons (e.g., numeration, whole 
number addition). A typical lesson began with one or two 
concept development activities that introduced the concept, 
typically in visual form. For example, age-appropriate 
animated graphic figures were used to demonstrate addition, 
number blocks and ten-sticks were manipulated to demon- 
strate place value, and matrices were shown to display 
multiplication concepts. Next, the student was provided 
directed practice containing high levels of prompts and cues, 
then automaticity practice followed, usually as a timed game 
activity. The lesson finished with an application activity 
(e.g., word problems, graphing, puzzle) that employed the 
concepts just learned. 

The reading software consisted of 1080 activities that 
provided approximately 216 hours of instruction from kin- 
dergarten through third grade levels. The curriculum util- 
ized a natural approach that emphasized understanding 
rather than memorization. For example, sounds were pre- 
sented in contexts of words, and words were presented in the 
contexts of sentences. Skills were directly taught (e.g., 
visual discrimination, letter recognition, consonants, vow- 
els, phonograms, decoding rules, sight words, vocabulary, 
and comprehension) but not in isolation. Nearly all words 
and sentences included on screen graphics of the word or 
sentence. 

The students attended computer lab for 30 minutes 
daily through the regular school year. Equal amounts of time 
were scheduled each week for each subject area. The teacher 
was present in the lab but was only minimally involved. The 
ILS software placed the students into different portions of 
each curriculum (Clariana, 1991) and the students pro- 
gressed through the standard ILS sequence at their own pace 
(Bond & Clariana, 1989). 

Posttests 

California Test of Basic Skills (CTBS/4) Mathematics 
Total and Reading Total scores served as posttest. Normal 
curve equivalent scores (NCE, an interval measure) rather 
than percentiles (NPR, an ordinal measure) were used for 
analysis. The test battery was given under regular classroom 
test conditions in the spring. 
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Results 



The design was a mixed multi- variate analysis of co- 
variance (MANCOVA), featuring one between-subjects 
factor (treatment: ILS versus control) and one within-sub- 
jects factor (content: reading and mathematics), two inde- 
pendent variables (third grade reading and mathematics 
scores), and two covariates (second grade reading and mathe- 
matics scores). Means and standard deviations are provided 
in Table 1. 



TABLE 1. Observed means (as NCEs) and Standard Devia- 
tions for Third Grade CTBS Reading and Mathematics tests. 



GROUP 




READING 


MATH 


Control (n = 38) 


Mean 


63.6 


50.6 




SD 


26.0 


23.9 


ILS (n= 47) 


Mean 


65.3 


62.4 


SD 


25.0 


24.9 



The treatment effect was significant, F( 1 ,8 1 ) = 12. 19, 
p = 0.001 (see Table 2). The effect of the within-subjects 
variable was not significant. An ordinal interaction between 
the ELS treatment and content was found, F( 1,81) = 5.21, 
p = 0.025 (see Figure 1). Reading scores were virtually 
identical for the control and ILS groups, but mathematics 
scores were dramatically higher for the ILS versus the con- 
trol group. 



TABLE 2. MANCOVA for CTBS Reading and Mathemat- 
ics Tests. 



socwci 




tfa 


Mi 


/ 




Between-subjects 












ILS/no ILS 


3918.386 


1 


3918.386 


12.193 


0.001 


Cov(R) 


20865.412 


1 


20865.412 


64.928 


0.000 


Cov(M) 


3319.660 


1 


3319.660 


10.330 


0.002 


error 


26030.328 


81 


321.362 






Within-subjects 












Content 


310.116 


1 


310.116 


1.425 


0.236 


Content * ILS 


1135.134 


1 


1135.134 


5.217 


0.025 


XX * Cov(R) 


2750.762 


1 


2750.762 


12.643 


0.001 


XX * Cov(M) 


2975.126 


1 


2975.126 


13.674 


0.000 


error 


17623.648 


81 


217.576 







Unadjusted effect sizes were calculated by taking the 
difference between the treatment and control groups and 
then dividing by the standard deviation of the control groups. 
Mathematics scores increased under ILS ( es = 0.49) but 
reading scores (es = 0.06) increased only minimally (ad- 
justed effect sizes for Mathematics 0.66, for Reading 0.20). 



O 





Figure 1. Interaction between treatment and content. 



Discussion 

The findings of this study confirmed that gains in 
mathematics scores were greater than gains in reading scores. 
The effect sizes obtained for this long duration ILS study 
were es = 0.49 for mathematics and es = 0.06 for reading. 
This amounts to a gain of about 2 NCEs in reading and 1 1 
NCEs in mathematics for the ILS group over the control 
group. 

Note that there are several factors that suggest that 
these effects sizes are under-reported: (a) the class size of the 
control group, 16 students per teacher, was 23% smaller than 
that of the ILS group, about 24 students per teacher. Thus the 
control should out-perform the ILS group, (b) Further, based 
on second grade tests, the control group was brighter than the 
ILS group, so their third grade scores should be greater. 

(c) Another vexing issue is that the ILS instruction did not 
perfectly correlate to the standardized test content, thus 
"watering down" larger potential gains. And finally, 

(d) these learning gains represent long-term retention. 

This study hypothesized that "good" CBI mathematics 
instruction may obtain larger gains than "good" CBI reading 
because the nature of mathematics content more closely 
corresponds to CBI media attributes. The findings of this 
study could be interpreted as supporting this hypothesis. 
Obviously, this requires more rigorous investigation. 

As indicated above, the teacher is a critical variable in 
longitudinal studies of achievement gains. Teacher expec- 
tations, either positive or negative, regarding the implemen- 
tation of an ILS (or CBI in general) may impact that teacher's 
behaviors before and during the implementation. For ex- 
ample, a teacher may intentionally "try harder" during the 
implementation of a new instructional method like ILS , with 
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a likely positive impact on learning performance. Alter- 
nately, if the teacher is negative or even indifferent, this 
response may transfer to students, with a likely negative 
impact on learning and thus on posttest performance. In this 
study, the second scenario is more likely than the first, 
though both probably occurred to some extent. 

Interestingly, follow-up interviews indicated that this 
teacher focused on reading during regular classroom in- 
struction both before and with the ILS. Emphasizing reading 
at the expense of mathematics (and other subjects) may be 
common in many elementary classrooms (Ball, 1988; Bums 
& Lash, 1988; Grouws & Good, 1988). The ILS instruction 
possibly compensated by providing an alternative coverage 

The ILS provided an alternative coverage 
of mathematics... 

of mathematics content In the same way, if a teacher 
focused on mathematics in the classroom, then an ILS may 
positively affect reading scores. 

In summary, no one instructional method is appropri- 
ate for every learner under every circumstance. An ILS, like 
any good instructional method, provides an alternative and 
supplement to classroom presentations. An ILS is only a part 
of the total learning environment but may change any or 
every aspect of this environment. ILS research should focus 
on how an ILS can change the total learning environment 
(i.e., curriculum coverage, interpersonal-interactions, moti- 
vation, remediation, parental involvement) and must con- 
sider teacher effects. 
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Appendix A - Data Set 



Group 


R2 


El 


M2 


Ml 












Control 


63 


57 


52 


40 


ILS 


33 


59 


31 


28 


Control 


53 


48 


88 


56 


ILS 


78 


60 


62 


52 


Control 


71 


55 


81 


73 


ILS 


63 


39 


94 


39 


Control 


95 


99 


94 


70 


ILS 


95 


98 


81 


89 


Control 


99 


89 


98 


89 


ILS 


67 


95 


81 


52 


Control 


63 


69 


99 


40 


ILS 


82 


97 


23 


47 


Control 


92 


85 


98 


75 


ILS 


55 


82 


74 


61 


Control 


95 


92 


99 


94 


ILS 


86 


65 


74 


83 


Control 


75 


73 


94 


77 


ILS 


44 


39 


49 


27 


Control 


35 


39 


13 


13 


ILS 


75 


39 


74 


75 


Control 


46 


43 


49 


24 


ILS 


61 


51 


37 


45 


Control 


89 


95 


71 


66 


ILS 


75 


33 


58 


54 


Control 


99 


73 


98 


75 


ILS 


93 


97 


81 


91 


Control 


98 


66 


76 


18 


ILS 


57 


70 


35 


36 


Control 


61 


70 


64 


75 


ILS 


53 


55 


81 


63 


Control 


85 


78 


62 


73 


ILS 


93 


99 


86 


95 


Control 


89 


59 


74 


58 


ILS 


61 


89 


89 


82 


Control 


85 


97 


81 


38 


ILS 


80 


82 


81 


53 


Control 


93 


89 


48 


40 


ILS 


89 


74 


89 


93 


Control 


21 


14 


51 


23 


ILS 


39 


29 


25 


30 


Control 


29 


9 


71 


26 


ILS 


80 


62 


64 


47 


Control 


79 


92 


74 


53 


ILS 


93 


66 


95 


91 


Control 


70 


59 


68 


43 


ILS 


89 


92 


68 


78 


Control 


43 


42 


43 


21 


ILS 


45 


74 


89 


93 


Control 


38 


47 


51 


34 


ILS 


35 


35 


46 


35 


Control 


85 


52 


95 


75 


ILS 


31 


22 


23 


26 


Control 


80 


92 


65 


75 


ILS 


85 


89 


84 


71 


Control 


79 


78 


30 


38 


ILS 


70 


92 


62 


95 


Control 


80 


95 


68 


69 


ILS 


75 


66 


91 


82 


Control 


80 


59 


81 


55 


ILS 


75 


74 


64 


10 


Control 


80 


89 


62 


61 


ILS 


98 


92 


97 


99 


Control 


75 


49 


68 


34 


ILS 


33 


59 


56 


63 


Control 


28 


29 


53 


2 


ILS 


99 


97 


98 


93 


Control 


12 


2 


13 


6 


ILS 


89 


92 


97 


89 


Control 


89 


62 


89 


73 


ILS 


70 


78 


51 


45 


Control 


28 


20 


43 


43 


ILS 


70 


97 


84 


75 


Control 


43 


78 


84 


61 












Control 


66 


74 


48 


38 












ILS 


86 


82 


74 


92 












ILS 


89 


85 


55 


89 












ILS 


61 


55 


71 


75 












ILS 


33 


24 


10 


56 












ILS 


24 


39 


55 


47 












ILS 


50 


65 


34 


40 












ILS 


23 


57 


5 


18 












ILS 


35 


27 


62 


77 












ILS 


55 


51 


88 


52 












ILS 


22 


10 


52 


25 












ILS 


50 


33 


46 


75 
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